Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPT

In this paper, we aimed to provide a review and tutorial for researchers in the field of medical imaging using language models to improve their tasks at hand. We began by providing an overview of the history and concepts of language models, with a special focus on large language models. We then reviewed the current literature on how language models are being used to improve medical imaging, emphasizing different applications such as image captioning, report generation, report classification, finding extraction, visual question answering, interpretable diagnosis, and more for various modalities and organs. The ChatGPT was specially highlighted for researchers to explore more potential applications. We covered the potential benefits of accurate and efficient language models for medical imaging analysis, including improving clinical workflow efficiency, reducing diagnostic errors, and assisting healthcare professionals in providing timely and accurate diagnoses. Overall, our goal was to bridge the gap between language models and medical imaging and inspire new ideas and innovations in this exciting area of research. We hope that this review paper will serve as a useful resource for researchers in this field and encourage further exploration of the possibilities of language models in medical imaging

Data-Driven Volumetric Image Generation from Surface Structures using a Patient-Specific Deep Leaning Model

The advent of computed tomography significantly improves patient health regarding diagnosis, prognosis, and treatment planning and verification. However, tomographic imaging escalates concomitant radiation doses to patients, inducing potential secondary cancer. We demonstrate the feasibility of a data-driven approach to synthesize volumetric images using patient surface images, which can be obtained from a zero-dose surface imaging system. This study includes 500 computed tomography (CT) image sets from 50 patients. Compared to the ground truth CT, the synthetic images result in the evaluation metric values of 26.9 Hounsfield units, 39.1dB, and 0.965 regarding the mean absolute error, peak signal-to-noise ratio, and structural similarity index measure. This approach provides a data integration solution that can potentially enable real-time imaging, which is free of radiation-induced risk and could be applied to image-guided medical procedures

Multi-dimension unified Swin Transformer for 3D Lesion Segmentation in Multiple Anatomical Locations

In oncology research, accurate 3D segmentation of lesions from CT scans is essential for the modeling of lesion growth kinetics. However, following the RECIST criteria, radiologists routinely only delineate each lesion on the axial slice showing the largest transverse area, and delineate a small number of lesions in 3D for research purposes. As a result, we have plenty of unlabeled 3D volumes and labeled 2D images, and scarce labeled 3D volumes, which makes training a deep-learning 3D segmentation model a challenging task. In this work, we propose a novel model, denoted a multi-dimension unified Swin transformer (MDU-ST), for 3D lesion segmentation. The MDU-ST consists of a Shifted-window transformer (Swin-transformer) encoder and a convolutional neural network (CNN) decoder, allowing it to adapt to 2D and 3D inputs and learn the corresponding semantic information in the same encoder. Based on this model, we introduce a three-stage framework: 1) leveraging large amount of unlabeled 3D lesion volumes through self-supervised pretext tasks to learn the underlying pattern of lesion anatomy in the Swin-transformer encoder; 2) fine-tune the Swin-transformer encoder to perform 2D lesion segmentation with 2D RECIST slices to learn slice-level segmentation information; 3) further fine-tune the Swin-transformer encoder to perform 3D lesion segmentation with labeled 3D volumes. The network's performance is evaluated by the Dice similarity coefficient (DSC) and Hausdorff distance (HD) using an internal 3D lesion dataset with 593 lesions extracted from multiple anatomical locations. The proposed MDU-ST demonstrates significant improvement over the competing models. The proposed method can be used to conduct automated 3D lesion segmentation to assist radiomics and tumor growth modeling studies. This paper has been accepted by the IEEE International Symposium on Biomedical Imaging (ISBI) 2023

Full-dose PET Synthesis from Low-dose PET Using High-efficiency Diffusion Denoising Probabilistic Model

To reduce the risks associated with ionizing radiation, a reduction of radiation exposure in PET imaging is needed. However, this leads to a detrimental effect on image contrast and quantification. High-quality PET images synthesized from low-dose data offer a solution to reduce radiation exposure. We introduce a diffusion-model-based approach for estimating full-dose PET images from low-dose ones: the PET Consistency Model (PET-CM) yielding synthetic quality comparable to state-of-the-art diffusion-based synthesis models, but with greater efficiency. There are two steps: a forward process that adds Gaussian noise to a full dose PET image at multiple timesteps, and a reverse diffusion process that employs a PET Shifted-window Vision Transformer (PET-VIT) network to learn the denoising procedure conditioned on the corresponding low-dose PETs. In PET-CM, the reverse process learns a consistency function for direct denoising of Gaussian noise to a clean full-dose PET. We evaluated the PET-CM in generating full-dose images using only 1/8 and 1/4 of the standard PET dose. Comparing 1/8 dose to full-dose images, PET-CM demonstrated impressive performance with normalized mean absolute error (NMAE) of 1.233+/-0.131%, peak signal-to-noise ratio (PSNR) of 33.915+/-0.933dB, structural similarity index (SSIM) of 0.964+/-0.009, and normalized cross-correlation (NCC) of 0.968+/-0.011, with an average generation time of 62 seconds per patient. This is a significant improvement compared to the state-of-the-art diffusion-based model with PET-CM reaching this result 12x faster. In the 1/4 dose to full-dose image experiments, PET-CM is also competitive, achieving an NMAE 1.058+/-0.092%, PSNR of 35.548+/-0.805dB, SSIM of 0.978+/-0.005, and NCC 0.981+/-0.007 The results indicate promising low-dose PET image quality improvements for clinical applications

Single energy CT-based mass density and relative stopping power estimation for proton therapy using deep learning method

BackgroundThe number of patients undergoing proton therapy has increased in recent years. Current treatment planning systems (TPS) calculate dose maps using three-dimensional (3D) maps of relative stopping power (RSP) and mass density. The patient-specific maps of RSP and mass density were obtained by translating the CT number (HU) acquired using single-energy computed tomography (SECT) with appropriate conversions and coefficients. The proton dose calculation uncertainty of this approach is 2.5%-3.5% plus 1 mm margin. SECT is the major clinical modality for proton therapy treatment planning. It would be intriguing to enhance proton dose calculation accuracy using a deep learning (DL) approach centered on SECT.ObjectivesThe purpose of this work is to develop a deep learning method to generate mass density and relative stopping power (RSP) maps based on clinical single-energy CT (SECT) data for proton dose calculation in proton therapy treatment.MethodsArtificial neural networks (ANN), fully convolutional neural networks (FCNN), and residual neural networks (ResNet) were used to learn the correlation between voxel-specific mass density, RSP, and SECT CT number (HU). A stoichiometric calibration method based on SECT data and an empirical model based on dual-energy CT (DECT) images were chosen as reference models to evaluate the performance of deep learning neural networks. SECT images of a CIRS 062M electron density phantom were used as the training dataset for deep learning models. CIRS anthropomorphic M701 and M702 phantoms were used to test the performance of deep learning models.ResultsFor M701, the mean absolute percentage errors (MAPE) of the mass density map by FCNN are 0.39%, 0.92%, 0.68%, 0.92%, and 1.57% on the brain, spinal cord, soft tissue, bone, and lung, respectively, whereas with the SECT stoichiometric method, they are 0.99%, 2.34%, 1.87%, 2.90%, and 12.96%. For RSP maps, the MAPE of FCNN on M701 are 0.85%, 2.32%, 0.75%, 1.22%, and 1.25%, whereas with the SECT reference model, they are 0.95%, 2.61%, 2.08%, 7.74%, and 8.62%. ConclusionThe results show that deep learning neural networks have the potential to generate accurate voxel-specific material property information, which can be used to improve the accuracy of proton dose calculation.Advances in knowledgeDeep learning-based frameworks are proposed to estimate material mass density and RSP from SECT with improved accuracy compared with conventional methods

Delineating the molecular landscape of different histopathological growth patterns in colorectal cancer liver metastases

BackgroundHistopathological growth patterns (HGPs) have shown important prognostic values for patients with colorectal cancer liver metastases, but the potential molecular mechanisms remain largely unknown.MethodsWe performed an exploratory analysis by conducting the RNA sequencing of primary colorectal lesions, colorectal liver metastatic lesions and normal liver tissues.FindingsWe found that desmoplastic HGPs of the metastatic lesions were significantly enriched in EMT, angiogenesis, stroma, and immune signaling pathways, while replacement HGPs were enriched in metabolism, cell cycle, and DNA damage repair pathways. With the exception of immune-related genes, the differentially expressed genes of the two HGPs from colorectal liver metastases were mostly inherited from the primary tumor. Moreover, normal liver tissue in the desmoplastic HGP subgroup was markedly enriched in the fibrinous inflammation pathway.ConclusionsWe surmised that HGPs are observable morphological changes resulting from the regulation of molecular expressions, which is the combined effect of the heterogeneity and remodeling of primary tumors seeds and liver soils

Distributed Coordination Control of First- and Second-Order Multiagent Systems with External Disturbances

This paper is devoted to the coordination control problem of heterogeneous first- and second-order multiagent systems with external disturbances. First, by applying the theory of eigenvalue and the method of model transformation, the consensus state of heterogeneous multiagent systems is obtained. Then, based on the consensus state, the control output is defined, and sufficient conditions are derived to make all agents reach consensus with H∞ performance. Finally, simulation results are provided to demonstrate the effectiveness of the presented results

BRD4 PROTAC degrader MZ1 exhibits anti-B-cell acute lymphoblastic leukemia effects via targeting CCND3

ABSTRACTIntroduction B-cell acute lymphoblastic leukemia (B-ALL) is the most prevalent malignant tumor affecting children. While the majority of B-ALL patients (90%) experience successful recovery, early relapse cases of B-ALL continue to exhibit high mortality rates. MZ1, a novel inhibitor of Bromodomains and extra-terminal (BET) proteins, has demonstrated potent antitumor activity against hematological malignancies. The objective of this study was to examine the role and therapeutic potential of MZ1 in the treatment of B-ALL.Methods In order to ascertain the fundamental mechanism of MZ1, a sequence of in vitro assays was conducted on B-ALL cell lines, encompassing Cell Counting Kit 8 (CCK8) assay, Propidium iodide (PI) staining, and Annexin V/PI staining. Western blotting and quantitative real-time polymerase chain reaction (qRT-PCR) were employed to examine protein and mRNA expression levels. Transcriptomic RNA sequencing (RNA-seq) was utilized to screen the target genes of MZ1, and lentiviral transfection was employed to establish stably-expressing/knockdown cell lines.Results MZ1 has been observed to induce the degradation of Bromodomain Containing 4 (BRD4), Bromodomain Containing 3 (BRD3), and Bromodomain Containing 2 (BRD2) in B-ALL cell strains, leading to inhibited cell growth and induction of cell apoptosis and cycle arrest in vitro. These findings suggest that MZ1 exhibits cytotoxic effects on two distinct molecular subtypes of B-ALL, namely 697 (TCF3/PBX1) and RS4;11 (MLL-AF4) B-ALL cell lines. Additionally, RNA-sequencing analysis revealed that MZ1 significantly downregulated the expression of Cyclin D3 (CCND3) gene in B-ALL cell lines, which in turn promoted cell apoptosis, blocked cell cycle, and caused cell proliferation inhibition.Conclusion Our results suggest that MZ1 has potential anti-B-ALL effects and might be a novel therapeutic target